Test Driven Development (TDD): 2008 Open Research

This open research into testing and test-driven development (TDD) was performed the second week of October 2008 and there was 121 respondents. The survey was announced on the Extreme Programming (XP) and Test-Driven Development (TDD) mailing lists and summarized in Test-Driven Development (TDD): Reality over Rhetoric. The goal was to find out what agile developers were actually doing to compare it with what’s being talked about. The survey results Comments Downloads What you may do with the information Links to other articles/surveys Why share this much information?

The Survey Results

Some findings include:

• Within the TDD/XP community there is a significant focus on other testing and validation techniques, as you can see in Figure 1.
• Even though there is a clear bias towards TDD on these lists, TDD is clearly not the only validation technique that has been commonly adopted by agile developers.
• Developer TDD is more common than acceptance TDD, perhaps because of the more technical focus within this community.
• For both types of TDD, pairing with an experienced person and mentoring were rated as the most effective means of learning the technique. Training came in a distant third followed by other techniques such as pairing with another learner, reading, and online forum discussions.
• Although many people are doing acceptance TDD, it is still more common for them to capture requirements specifications in the form of documents or models (see Figure 2).
• Although many people are doing developer TDD, it is still more common for them to capture design specifications in the form of documents or models (see Figure 3).
• The fact that the survey found that agile development teams are writing documentation is consistent with findings from other surveys, in particular DDJ’s 2008 Modeling and Documentation survey which specifically investigated this topic.
• Some people are using CASE tools (despite rhetoric to the contrary) for requirements and design capture.
• Inclusive models (using whiteboards or paper) are more common than more complex models captured using software-based modeling tools (CASE tools).

Figure 1. Testing/Validation practices on agile teams.

Figure 2. Requirements capture practices on agile teams.

Figure 3. Design capture practices on agile teams.

Comments

• The people surveyed, subscribers to the XP and TDD lists. are the ones who are most likely to be doing test-driven development (TDD), so it wouldn’t be appropriate to consider the adoption rate of TDD of this study to be anywhere near the adoption rate across the industry.
• I think that the results do reflect general practice within the agile community, which is interesting because there is nowhere near as much discussion of requirements and design documentation as there is about executable tests as specification.
• I was disappointed with the response rate of 121.
• This survey suffers from the fundamental challenges faced by all surveys.

Downloads

The Survey Questions(70 K)

Raw Data(141 K)

Summary Presentation(274 K)

What You May Do With This Information

You may use this data as you see fit, but may not sell it in whole or in part. You may publish summaries of the findings, but if you do so you must reference the survey accordingly (include the name and the URL to this page). Feel free to contact me with questions. Better yet, if you publish, please let me know so I can link to your work.

Links to Other Articles/Surveys

1. My other surveys

Why Share This Much Information?

I’m sharing the results, and in particular the source data, of my surveys for several reasons:

1. Other people can do a much better job of analysis than I can. If they publish online, I am more than happy to include links to their articles/papers.
2. Once I’ve published my column summarizing the data in DDJ, I really don’t have any reason not to share the information.
3. Too many traditionalists out there like to use the “where’s the proof” question as an excuse not to adopt agile techniques. By providing some evidence that a wide range of organizations seem to be adopting these techniques maybe we can get them to rethink things a bit.
4. I think that it’s a good thing to do and I invite others to do the same.

Suggested Reading

This is an eye-opening book for anyone who is trying to understand how to measure concepts, such as developer productivity levels for example, which are often perceived as difficult to measure. If you choose to think outside of the metrics box for a bit you’ll quickly realize that you can easily measure information which is critical to your decision making process.